Reinforcement Learning Benchmarks and Bake-offs II

Evolution of neural networks, through genetic algorithms or otherwise, hasrecently emerged as a possible way to solve challenging reinforcement learningproblems; NEAT (or NeuroEvolution of Augmenting Topologies), is a particularlypowerful such method. Because it is based on evolving a population of solutions,where individuals are evaluated on a number of test episodes, the current setup ofthe RL Benchmarking Event tasks makes it impossible to compare with it directly.In particular, neuroevolution methods are geared towards finding the best solution(instead of optimizing average reward over the course of learning), and towardsreinforcement over the entire episode (instead of during the episode). NEAT inparticular is best suited for challenging tasks where topology evolution has time tohave an effect, especially those that are non-Markovian and therefore require recur-rency. Nevertheless, it is instructive to compare how NEAT solves these problems,and where the particular strengths and weaknesses are. 1 The NEAT Algorithm The NeuroEvolution of Augmenting Topologies (NEAT) method [6] is a policy-searchreinforcement learning algorithm that uses a genetic algorithm to search for optimalneural network policies. NEAT automatically evolves network topology to fit the com-plexity of the problem by combining the usual search for the appropriate networkweights with complexification of the network structure. By starting with simple net-works and expanding the search space only when beneficial, NEAT is able to findsignificantly more complex controllers than other fixed-topology learning algorithms.This approach is highly effective: NEAT outperforms other neuroevolution (NE) meth-ods on complex control tasks like the double pole balancing task [5, 6] and the roboticstrategy-learning domain [7]. These properties make NEAT an attractive method forevolving neural networks in complex tasks. In this section, the NEAT method is brieflyreviewed; see [5, 6, 7] for more detailed descriptions.NEAT is based on three key ideas. First, evolving network structure requires a flex-ible genetic encoding. Each genome in NEAT includes a list of connection genes, eachof which refers to two node genes being connected. Each connection gene specifiesthe in-node, the out-node, the weight of the connection, whether or not the connec-